<html>
<head><meta charset="utf-8"><title>Absolute performance numbers · t-compiler/performance · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/index.html">t-compiler/performance</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html">Absolute performance numbers</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="246857150"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/246857150" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#246857150">(Jul 22 2021 at 14:47)</a>:</h4>
<p><span class="user-mention" data-user-id="125250">@Wesley Wiser</span> <span class="user-mention" data-user-id="116083">@pnkfelix</span> not sure I follow your logic on <a href="https://perf.rust-lang.org/compare.html?start=b5a2ccee81406303324016d03399fac68ceb6718&amp;end=7d3788b67103a424bf6612495afedb80c4fe8b10">this perf regression</a>. Absolute regression numbers can be useful, but we provide regression in percentage terms for a reason. I agree that for benchmarks that are stress tests and not real world crates, we should probably be a bit more lenient, but an 8% regression isn't something we should just shrug off IMO.</p>



<a name="246857750"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/246857750" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> The 8472 <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#246857750">(Jul 22 2021 at 14:51)</a>:</h4>
<p>For <code>cargo watch -x check -x test</code> style of uses we also want small values to stay small, not just shave off time from the big ones.</p>



<a name="246857756"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/246857756" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#246857756">(Jul 22 2021 at 14:51)</a>:</h4>
<p>I agree -- saying things like regressions are in the millisecond range is usually not terribly helpful, because a <em>lot</em> of our benchmarks (intentionally!) take less than a second to run. This means we can collect more data per run on different kinds of builds, but it does mean timing is going to really matter. If your benchmark takes 300ms, then 1ms is actually pretty big.</p>



<a name="246861238"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/246861238" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#246861238">(Jul 22 2021 at 15:13)</a>:</h4>
<p>Hmm that's an interesting point. I think for the specific issue we're discussing, it's still worth landing with the regression. All of the real world crates (including hello-world) show &lt; 1% regression and some are even showing performance improvements with the exception of <code>wg-grammar</code>.</p>



<a name="246862052"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/246862052" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#246862052">(Jul 22 2021 at 15:20)</a>:</h4>
<p>I just realized part of the disconnect here might be that in last week's compiler team meeting, many of us saw the 20% regression on the <code>deeply-nested-check</code> benchmark and that framed the discussion while <span class="user-mention" data-user-id="224872">@rylev</span> you're talking about the 8% regression on the <code>-debug</code> variant. I think that's a much more reasonable number than the 20% we saw originally.</p>



<a name="246862143"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/246862143" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#246862143">(Jul 22 2021 at 15:21)</a>:</h4>
<p>Perhaps all I'm saying is that it would be helpful if tests had an indication when they run for an extremely small amount of time (like &lt; 50ms or something).</p>



<a name="246862190"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/246862190" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#246862190">(Jul 22 2021 at 15:21)</a>:</h4>
<p>You can look at the wall-time metrics to get that</p>



<a name="246862205"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/246862205" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#246862205">(Jul 22 2021 at 15:21)</a>:</h4>
<p>but, like, instructions are used for a reason</p>



<a name="246863815"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/246863815" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#246863815">(Jul 22 2021 at 15:34)</a>:</h4>
<p>Yeah, that's definitely true.</p>



<a name="246866697"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/246866697" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#246866697">(Jul 22 2021 at 15:56)</a>:</h4>
<p>The real world crates we have are not indicative of all real world usage. An 8% regression in a stress test benchmark could potentially lead to a noticeable regressions in real world crates - maybe not 8%, but who’s to say? </p>
<p>In general, if we’re going to ignore 8% regressions in benchmarks, we should be asking why we have that benchmark in the first place.</p>



<a name="246867077"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/246867077" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#246867077">(Jul 22 2021 at 15:58)</a>:</h4>
<p>On the other hand, an acceptable justification could be “despite 8% regressions in some benchmarks, the combination of the fact that other benchmarks improved along with the utility of the change itself, makes it acceptable for there to be an 8% regression”.</p>



<a name="246867139"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/246867139" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#246867139">(Jul 22 2021 at 15:59)</a>:</h4>
<p>But we should probably talk over some of this and come up with better guides for what is acceptable and what is not.</p>



<a name="246867161"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/246867161" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#246867161">(Jul 22 2021 at 15:59)</a>:</h4>
<p>is this the correct link to the perf results btw ? isn't this PR a correctness fix for the incremental fingerprint errors we're trying to fix since march and almost have to accept the regressions <span aria-label="sweat" class="emoji emoji-1f613" role="img" title="sweat">:sweat:</span>  ?</p>



<a name="247287179"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247287179" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247287179">(Jul 27 2021 at 02:24)</a>:</h4>
<p>I’m mostly in <span class="user-mention" data-user-id="116113">@lqd</span> ’s camp here. The real justification for letting that PR through is that it seems like the best option available for resolving a particular subclass of fingerprint issues</p>



<a name="247287206"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247287206" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247287206">(Jul 27 2021 at 02:24)</a>:</h4>
<p>We can work on figuring out ways to address the performance hit over time</p>



<a name="247287416"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247287416" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247287416">(Jul 27 2021 at 02:27)</a>:</h4>
<p>Also, there’s another way of framing the discussion, at least for the changes that are fixing fingerprint issues: As long as the fingerprint issues remain significant, people will be forced to turn incr-comp off (or lazily bypass it via <code>cargo clean</code>, etc). So for every PR that addresses a fingerprint issue, we should be focusing <strong>at most</strong> on the performance hit for “full” build types. All the <code>incr-*</code> variants should be heavily discounted, because if fingerprint bugs persist, then incr-comp is turned off (at least for some people), and as long as incr-comp is turned off, then the only perfomance regressions that matter are those that hit “full” build types.</p>



<a name="247287637"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247287637" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247287637">(Jul 27 2021 at 02:30)</a>:</h4>
<p>(well, okay, actually: After I narrowed the UI to just the “full” build types, it didn’t decrease the degree nor number of regressing benchmarks significantly. So that particular argument doesn’t really apply here; sorry for the noise.)</p>



<a name="247298459"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247298459" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247298459">(Jul 27 2021 at 06:28)</a>:</h4>
<p>btw <a href="https://github.com/rust-lang/rust/pull/84944">https://github.com/rust-lang/rust/pull/84944</a> improves performance on a bunch of the bigger benchmark regressions seen in the "correctness fix" results above</p>



<a name="247308106"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247308106" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247308106">(Jul 27 2021 at 08:45)</a>:</h4>
<p><span class="user-mention" data-user-id="116083">@pnkfelix</span> <span class="user-mention" data-user-id="116113">@lqd</span> to be clear the issue at hand here is not the performance regression itself (I agree with your assessment that the fix is worth the regression) - it’s the lack of written justification. Your explanation above is not documented in the PR at all, and one just has to know about all this context to understand why the performance regression is acceptable.</p>



<a name="247308267"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247308267" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lqd <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247308267">(Jul 27 2021 at 08:47)</a>:</h4>
<p>aaah thanks for the clarification (it wasn't obvious to me that this was the issue, rather than the topic of absolute numbers on this PR's results, sorry about that), that makes sense</p>



<a name="247308410"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247308410" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247308410">(Jul 27 2021 at 08:49)</a>:</h4>
<p>Well, the justification given was “this actually isn’t a performance regression” which I don’t agree with. This was a performance regression, just one that’s worth the cost.</p>



<a name="247337115"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247337115" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247337115">(Jul 27 2021 at 14:02)</a>:</h4>
<p><span class="user-mention silent" data-user-id="224872">rylev</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/Absolute.20performance.20numbers/near/247308410">said</a>:</p>
<blockquote>
<p>Well, the justification given was “this actually isn’t a performance regression” which I don’t agree with. This was a performance regression, just one that’s worth the cost.</p>
</blockquote>
<p>Can you point to where that statement was made? The only <a href="https://github.com/rust-lang/rust/pull/85868#issuecomment-880760352">comment I can find</a> says "This performance is worse than the two other approaches […], but this is less hacky, and more clearly correct."</p>



<a name="247342561"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247342561" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247342561">(Jul 27 2021 at 14:43)</a>:</h4>
<p>This was spurred originally by <a href="#narrow/stream/238009-t-compiler.2Fmeetings/topic/.5Bweekly.5D.202021-07-22.20.2354818">this comment in zulip</a> not by the pull request. My characterization is perhaps slightly unfair, but the gist is that the comment made was not of the nature "the regression is acceptable because the fix is necessary" but rather "the performance was not so bad so let's just merge this".</p>



<a name="247342947"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247342947" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247342947">(Jul 27 2021 at 14:46)</a>:</h4>
<p>I think we need to do a better job of having guidance for characterizing how bad a perf regression is. We're pretty good now at knowing whether a regression has occurred or not but not what the actual magnitude of the regression is. This is trickier because the percentage regression means different things depending on the benchmark in question. For example, stress test benchmarks are likelier to show large regressions when they regress (because they typically hit the same code paths over and over) while real world crates might have smaller regressions since their code exercises more code paths than stress tests do (typically).</p>



<a name="247343065"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247343065" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247343065">(Jul 27 2021 at 14:47)</a>:</h4>
<p>Once we have a mechanism for quantifying the severity of a regression, it will be easier to say things like "regression is X bad, but weighed against the importance of the PR in question, an X bad regression is worth it"</p>



<a name="247356372"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247356372" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247356372">(Jul 27 2021 at 16:23)</a>:</h4>
<p><span class="user-mention silent" data-user-id="224872">rylev</span> <a href="#narrow/stream/247081-t-compiler.2Fperformance/topic/Absolute.20performance.20numbers/near/247342561">said</a>:</p>
<blockquote>
<p>This was spurred originally by <a href="#narrow/stream/238009-t-compiler.2Fmeetings/topic/.5Bweekly.5D.202021-07-22.20.2354818">this comment in zulip</a> not by the pull request. My characterization is perhaps slightly unfair, but the gist is that the comment made was not of the nature "the regression is acceptable because the fix is necessary" but rather "the performance was not so bad so let's just merge this".</p>
</blockquote>
<p><span class="user-mention" data-user-id="224872">@rylev</span> you probably meant <a href="#narrow/stream/238009-t-compiler.2Fmeetings/topic/.5Bweekly.5D.202021-07-22.20.2354818/near/246856159">this comment</a>, right? (Zulip links are non-trivial. <span aria-label="sad" class="emoji emoji-2639" role="img" title="sad">:sad:</span> )</p>



<a name="247357838"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247357838" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> rylev <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247357838">(Jul 27 2021 at 16:35)</a>:</h4>
<p>Ah yes sorry!</p>



<a name="247358438"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/247081-t-compiler/performance/topic/Absolute%20performance%20numbers/near/247358438" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/247081-t-compiler/performance/topic/Absolute.20performance.20numbers.html#247358438">(Jul 27 2021 at 16:41)</a>:</h4>
<p>okay that helps a lot to provide context of conversation</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>